Skip to content

fixed conversion to sparse for non-numeric index #11856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

DSLituiev
Copy link

this fixes #11633

this fix breaks some sparse tests,
stuck on test_combine_first : ValueError: total size of new array must be unchanged

@jreback jreback added Sparse Sparse Data Type Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Dec 16, 2015
@jreback
Copy link
Contributor

jreback commented Jan 7, 2016

this looks reasonable. I think you may have to adjust some tests, and maybe some more fixing of the dtype propogation.

@DSLituiev
Copy link
Author

I fixed one test to its original state (accounting for conversion of integer values to floats with to_sparse). Now ./test_sparse.py runs OK.

@jreback
Copy link
Contributor

jreback commented Feb 12, 2016

can you rebase when you have a chance.

@DSLituiev DSLituiev force-pushed the sparse_nonnumeric_index branch from 8283a33 to 177355e Compare February 20, 2016 22:33
@DSLituiev
Copy link
Author

rebased

@jreback
Copy link
Contributor

jreback commented Feb 23, 2016

need some tests for this.

@DSLituiev
Copy link
Author

I added a test.
One existing test fails now.

@@ -1720,6 +1720,17 @@ def test_nan_columnname(self):
nan_colname_sparse = nan_colname.to_sparse()
self.assertTrue(np.isnan(nan_colname_sparse.columns[0]))

def test_pivot(self):
df = pd.DataFrame( list(zip([3,2,4,1,5,3,2],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the issue number as a comment; don't use pd.

@jreback jreback added this to the 0.18.1 milestone Feb 23, 2016
@DSLituiev DSLituiev force-pushed the sparse_nonnumeric_index branch from 852838d to 8bc8f28 Compare February 23, 2016 19:00
@DSLituiev
Copy link
Author

The current fix still does not completely solve the problem of sparse pivot #11633 (it uses sparse as input, but produces a dense output). One will need to look into pivot_table itself to see if one can make it spit out a sparse table.

@jreback
Copy link
Contributor

jreback commented Feb 23, 2016

@DSLituiev that's ok, let's get this bug fix in, then can work on that (which is a bit more complicated). you can always start constructing a fix based on THIS branch if you would like.

@sinhrks
Copy link
Member

sinhrks commented Apr 5, 2016

@DSLituiev can you update?

@jreback
Copy link
Contributor

jreback commented Apr 9, 2016

@DSLituiev can you update, lots of Sparse changes recently.

@DSLituiev
Copy link
Author

I'll have a look

On Sat, Apr 9, 2016 at 8:05 AM, Jeff Reback [email protected]
wrote:

@DSLituiev https://github.com/DSLituiev can you update, lots of Sparse
changes recently.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#11856 (comment)

@DSLituiev DSLituiev force-pushed the sparse_nonnumeric_index branch from 8bc8f28 to 990c073 Compare April 9, 2016 19:57
@jreback
Copy link
Contributor

jreback commented Apr 18, 2016

can you rebase/update

@jreback jreback modified the milestones: 0.18.2, 0.18.1 Apr 18, 2016
@jreback
Copy link
Contributor

jreback commented May 7, 2016

can you rebase / update, we've had lots of sparse changes.

@sinhrks
Copy link
Member

sinhrks commented May 17, 2016

As it is important for #667, I've prepared #13201.

@jreback jreback added this to the 0.18.2 milestone May 17, 2016
@jreback
Copy link
Contributor

jreback commented May 17, 2016

superseded by #13021

@jreback jreback closed this May 17, 2016
jreback pushed a commit that referenced this pull request May 18, 2016
closes #11633
closes #11856

Author: sinhrks <[email protected]>

Closes #13201 from sinhrks/sparse_isnull and squashes the following commits:

443b47e [sinhrks] BUG: Sparse creation with object dtype may raise TypeError
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Sparse Sparse Data Type
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pivot to SparseDataFrame: TypeError: ufunc 'isnan' not supported in sparse matrix conversion
3 participants